{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 12 - Simulations and hypotheses\n",
    "\n",
    "We will start this lab by simulating some data using the [NumPy](http://www.numpy.org) package.  The NumPy package is used by Pandas, so should have been installed when we installed Pandas.  However, we still need to import it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The smoking rate in New York City was 11.5% in 2016.  This statistic means that in theory if we picked 100 New Yorkers at random, we expect 11.6 of them to smoke.  But what happens in practice if we select 100 random New Yorkers?  We are going to simulate this scenario using code.\n",
    "\n",
    "First we define our population:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "population = [\"Smoker\",\"Non-smoker\"]\n",
    "pop_prob = [0.116,1-0.116]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code defines our population as having two groups `Smoker` and `Non-smoker`, with the probabilities 0.116 and 1-0.116 = 0.884, respectively.  \n",
    "\n",
    "We can generate a random sample of 100 people from our population with the code `np.random.choice(population,p=pop_prob,size=100)`.  Type and run it below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code has simulated 100 people and labeled each as a `Smoker` or `Non-smoker` according to the probabilities we gave.  Try re-running this code.  What happens?\n",
    "\n",
    "Also notice that the word `array` appears at the beginning of the output.  An array is similar to a list, in that it holds many different values of the same type (integers, strings, etc.). This particular array is an `ndarray`, which is one of the kinds of arrays produced by `NumPy`.  We have to convert the array into a Pandas Series before we can use any Pandas function on it.\n",
    "\n",
    "Next we want to count the number of smokers, which we do by:\n",
    "\n",
    "1. saving the array into a variable\n",
    "2. converting the array into a Pandas Series using `Pd.Series(name_of_array)`\n",
    "3. using value_counts() to count the number of smokers and non-smokers\n",
    "\n",
    "Can figure out how to write this code?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "    <code>sample_array = np.random.choice(population,p=pop_prob,size=100)\n",
    "pd.Series(sample_array).value_counts()</code>\n",
    "</details>\n",
    "\n",
    "To understand the variation in the number of smokers, we want to repeat the above sampling many times and make a histogram of the counts of smokers.  So we will need to extract the number of smokers from the `value_counts()` output. Modify your code above to save the `value_counts()` output as the variable `counts`. \n",
    "\n",
    "Then we can get the number of smokers with the code `counts[\"Smoker\"]`.  Try it below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Lab 10, we sampled multiple times from our dataset and made a histogram of the mean, median, or variance of the samples.  Here we want to simulate multiple samples and make a histogram of the counts of smokers.  Try to write this code below.\n",
    "\n",
    "<details> <summary>Hint (pseudo-code):</summary>\n",
    "    <code>initialize a list for the counts\n",
    "loop 200 times:\n",
    "        simulate a new sample\n",
    "        count the number of smokers in the sample\n",
    "        append the smoker count in the latest simulation to your list\n",
    "turn the list into a Pandas series and make a histogram of it</code>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Answer:</summary>\n",
    "    <code>\n",
    "all_counts = []\n",
    "for i in range(200):\n",
    "    sample_array = np.random.choice(population,p=pop_prob,size=100)\n",
    "    counts = pd.Series(sample_array).value_counts()\n",
    "    all_counts.append(counts[\"Smoker\"])\n",
    "pd.Series(all_counts).hist()</code>\n",
    "</details>\n",
    "\n",
    "What do you notice about the histogram?  Run your code again.  How does the histogram change (or not change)?\n",
    "\n",
    "What happens when you increase the number of simulations?\n",
    "\n",
    "In most high-income countries, the percentage of births that are boys is 51.2%.  Consider samples of 100 births.  How much variation is there in the number of boys born?  Generate 250 samples of size 100 and plot a histgram of the number of boys born in each sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hypotheses\n",
    "\n",
    "A *statistical hypothesis* is a specific, testable assumption about a population that is either true or false.  Next class we will see how to formally test a hypothesis.  For this lab, we will focus on making our hypotheses specific enough that we can get some kind of answer from the data.\n",
    "\n",
    "Suppose we want to know if taxis with more passengers take longer trips.  We can rephrase this as a statement that is either true or false:\n",
    "\n",
    "*Taxis with more passengers take longer trips.*\n",
    "\n",
    "Presumably we can look at trips with lots of passengers and trips with few passengers, and compare how long these trips are.  But what is the cut-off between between lots and few passengers?  What exactly should we compare: the mean trip length?  The median trip length?  Something else?\n",
    "\n",
    "We will encode our choices to the above questions in the hypothesis:\n",
    "\n",
    "*Taxis with more than 1 passenger take longer trips on average than taxis with more than 1 passenger.*\n",
    "\n",
    "Now we can test this hypothesis.  First load the green taxi trip dataset into a dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, make a dataframe containing only the green taxi trips with 1 passenger."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the mean trip distance for these green taxi trips with only 1 passenger?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, make a dataframe containing only green taxi trips with 2 or more passengers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the mean trip distance for the green taxi trips with 2 or more passengers?  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is there any difference between the mean length of taxi trips with 1 passenger and taxi trips with 2 or more passengers?  Do you think this difference is significant?\n",
    "\n",
    "#### Challenges:\n",
    "- Is there any difference between the mean length of taxi trips that begin at JFK or Newark airports?  (RateCodeID is 2 or 3) and the other taxi trips?\n",
    "- Is there any difference between the median length of taxi trips paid by credit card (Payment_type is 1) and trips paid by cash (Payment_type is 2)?\n",
    "- Is there any difference in the mean number of passengers in trips taken at night from those taken during the day?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}